Unsupervised Learning Project¶
Data Description:¶
The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
Domain:¶
Object recognition
Attribute Information:¶
- All the features are geometric features extracted from the silhouette.
- All are numeric in nature.
Objective:¶
Apply dimensionality reduction technique – PCA and train a model using principle components instead of training the model using just the raw data.
1. Import the Libraries¶
In [1]:
#Importing All the Libriaries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import copy
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
sns.set_style(style='darkgrid') # Setting the plot style. Purely for aesthetic purposes
%matplotlib inline
2. Load the dataset¶
In [2]:
#Read the data as 'Data' dataframe
Data = pd.read_csv('vehicle-1.csv')
In [3]:
#Reading the first five rows of dataset
Data.head()
Out[3]:
In [4]:
#last 5 rows using tail() function
Data.tail()
Out[4]:
- All the columns seem to be numeric, although not necesarily continuous
- Let's check the datatypes, shape and number of missing values in the dataframe to see if there are read as numerical attributes
3. Data Preprocessing¶
In [5]:
#checking the shape of dataset. Dataset having 846 rows and 19 columns.
Data.shape
Out[5]:
In [6]:
#To show the detailed summary
Data.info()
In [7]:
#class attribute is not an object it is a category
Data['class'] = Data['class'].astype('category')
In [8]:
#This is showing dataset having 3 classes, Car, bus and van.
Data['class'].value_counts()
Out[8]:
In [9]:
#To get the number of columns
Data.columns
Out[9]:
In [10]:
Data.describe().T
Out[10]:
It gives the descriptive statistics (mean, median, mode, percentiles, min, max, standard deviation) of the columns of the dataset. By analysing it, we can see that:
compactness, circularity, distance_circularity, elongatedness, pr.axis_rectangularity, max.length_rectangularity, scaled_radius_of_gyration, scaled_radius_of_gyration.1, skewness_about.2, hollows_ratio are approximately normally distributed.
radius_ratio, pr.axis_aspect_ratio, max.length_aspect_ratio, scatter_ratio, scaled_variance, scaled_variance.1, skewness_about, skewness_about.1 are approx. right skewed distribution.
Checking Missing Value¶
In [11]:
# Checking Missing value present in the dataset. It seems that datset is having missing value
Data.apply(lambda x : sum(x.isnull()))
Out[11]:
In [12]:
#replacing blank value by 'NAN'
Data = Data.replace(' ', np.nan)
Data.head(10)
Out[12]:
Handling Missing Value¶
In [13]:
#Replacing the missing values by median
for i in Data.columns[:17]:
median_value = Data[i].median()
Data[i] = Data[i].fillna(median_value)
- As some data are rightly skewed so we will use median to handle the missing values.
In [14]:
#Checking the dataset again for missing value. It seems that all missing value are replaced with median.
Data.apply(lambda x : sum(x.isnull()))
Out[14]:
Understanding the outier using boxplot¶
In [15]:
Data.boxplot(figsize =(35,15))
Out[15]:
- It is showing that there are some columns which contains outliers such as radius_ratio, pr.axis_aspect_ratio, max.length_aspect_ratio, scaled_variance, scaled_variance.1, skewness_about, skewness_about.1.
In [16]:
#find the outliers and replace them by median
for column in Data.columns:
if column in ['radius_ratio','pr.axis_aspect_ratio', 'max.length_aspect_ratio','scaled_variance', 'scaled_variance.1','scaled_radius_of_gyration.1','skewness_about', 'skewness_about.1']:
q1 = Data[column].quantile(0.25)
q3 = Data[column].quantile(0.75)
iqr = q3 - q1
high = float(q3 + 1.5 * iqr)
low = float(q1 - 1.5 * iqr)
Data.loc[(Data[column]<low) | (Data[column] > high), column] = Data[column].median()
Data.plot(kind = 'box', figsize = (30,20))
Out[16]:
In [17]:
#Showing Data distribution using Histogram
Data.hist(figsize=(15,15))
Out[17]:
In [18]:
#Data Distribution using KDE
plt.figure(figsize= (30,20)) # Set the figure size
pos = 1 # a variable to manage the position of the subplot in the overall plot
for feature in Data.columns: # for-loop to iterate over every attribute whose distribution is to be visualized
plt.subplot(6, 4, pos) # plot grid
if feature in ['compactness', 'circularity', 'distance_circularity', 'radius_ratio','pr.axis_aspect_ratio', 'max.length_aspect_ratio', 'scatter_ratio', 'elongatedness', 'pr.axis_rectangularity', 'max.length_rectangularity','scaled_variance', 'scaled_variance.1', 'scaled_radius_of_gyration','scaled_radius_of_gyration.1', 'skewness_about', 'skewness_about.1','skewness_about.2', 'hollows_ratio']: # Plot histogram for all the continuous columns
sns.distplot(Data[feature], kde= True, kde_kws={'bw': 0.1} )
else:
sns.countplot(Data[feature], palette= 'Blues') # Plot bar chart for all the categorical columns
pos += 1 # to plot over the grid one by one
- From The above observation we can see data distribution of each variable.
- Most of the variables data are left and right skewed.
4. Understanding the Attributes¶
Dependant Atrribute¶
In [19]:
#Class is dependant attribute and having 3 unique categories as Car, bus and van.
Data['class'].value_counts()
Out[19]:
In [20]:
sns.countplot(Data['class'])
Out[20]:
In [21]:
#Encoding of categorical variables
from sklearn.preprocessing import LabelEncoder
labelencoder_X=LabelEncoder()
Data['class']=labelencoder_X.fit_transform(Data['class'])
Independant Attributes¶
In [22]:
#Correlation Matrix
corr = Data.corr()
corr
Out[22]:
In [23]:
#Showing correlation between each variable using heatmap
corr = abs(Data.corr()) # correlation matrix
lower_triangle = np.tril(corr, k = -1) # select only the lower triangle of the correlation matrix
mask = lower_triangle == 0 # to mask the upper triangle in the following heatmap
plt.figure(figsize = (20,10)) # setting the figure size
sns.set_style(style = 'white') # Setting it to white so that we do not see the grid lines
sns.heatmap(lower_triangle, center=0.5, cmap= 'Blues', annot= True, xticklabels = corr.index, yticklabels = corr.columns,
cbar= False, linewidths= 1, mask = mask) # Da Heatmap
plt.xticks(rotation = 50) # Aesthetic purposes
plt.yticks(rotation = 20) # Aesthetic purposes
plt.show()
Form the above obesevation:
- max_lengh_rectangularity, scaled_radiaus_of_gyration are highly corelated with circularity.
- scatter_ratio,elongatedness are corelated with distance_circularity.
- elongatedness, pr.axis_rectangularity, scaled_variance,scaled_variance.1 are higly correlated with scatter_ratio.
- pr.axis_rectangularity, scaled_variance,scaled_variance.1 are higly correlated with elongatedness.
- scaled_variance,scaled_variance.1 are higly correlated with pr.axis_rectangularity.
- scaled_variance.1 are higly correlated with scaled_variance.
- scaled_radiaus_of_gyration are corelated with hollows_ratio.
In [24]:
sns.pairplot(Data, hue = 'class', diag_kind='kde') # pairplot
plt.show()
- compactness has positive linear relationship with circularity, distance_circularity, radius_ratio, scatter_ratio, pr.axis_rectangularity, max.axis_rectangularity, scaled_variance1, scaled_variance. compactness has negative linear relationship with elongatedness.
- circularity has positive linear relationship with distance_circularity, scatter_ratio, pr.axis_rectangularity, max.axis_rectangularity, scaled_variance, scaled_variance1, scaled_radius_of_gyration. circularity has negative linear relationship with elongatedness.
- distance_circularity has positive linear relationship with radius_ratio, scatter_ratio, pr.axis_rectangularity, max.axis_rectangularity, scaled_variance, scaled_variance1, scaled_radius_of_gyration. distance_circularity has negative linear relationship with elongatedness.
- radius_ratio has positive linear relationship with pr.axis_aspect_ratio, scatter_ratio, scaled_variance, scaled_variance1, scaled_radius_of_gyration. radius_ratio has negative linear relationship with elongatedness.
In [25]:
Data.head()
Out[25]:
5. Without applying Dimensionality Reduction¶
In [26]:
#Now separate the dataframe into dependent and independent variables
X = Data.iloc[:,0:18].values
y = Data.iloc[:,18].values
In [27]:
from sklearn.preprocessing import StandardScaler
sc =StandardScaler()
X = sc.fit_transform(X)
In [28]:
#now split the data into 70:30 ratio
#orginal Data
Orig_X_train,Orig_X_test,Orig_y_train,Orig_y_test = train_test_split(X,y,test_size=0.30,random_state=4)
In [29]:
# Since we will be using multiple models using the same data, thhe following is a user defined function for that purpose
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import multilabel_confusion_matrix
def fit_n_print(model, Orig_X_train, Orig_X_test, Orig_y_train, Orig_y_test): # take the model, and data as inputs
from sklearn import metrics
model.fit(Orig_X_train, Orig_y_train) # fir the model with the train data
Orig_pred_train = model.predict(Orig_X_train) # model predictions on the train data
Orig_pred = model.predict(Orig_X_test) # make predictions on the test set
score = round(model.score(Orig_X_test, Orig_y_test), 4) # compute accuracy score for test set
#Printing Confusion Matrix
cm = metrics.confusion_matrix(Orig_pred,Orig_y_test, labels = [0,1,2])
df_cm = pd.DataFrame(cm, index = [i for i in ["Van ", "Car ", "Bus"]],
columns = [i for i in ["Van ", "Car ", "Bus"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g')
print(classification_report(Orig_y_test,Orig_pred))
recall = metrics.recall_score(Orig_y_test, Orig_pred, average='macro')
precision = metrics.precision_score(Orig_y_test, Orig_pred, average='macro')
f1_score = round(2*precision*recall/(precision + recall),3)
return score, recall, precision, f1_score # return all the metrics
1. LogisticRegression¶
In [30]:
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import BernoulliNB
# Fitting Logistic Regression to the Training set
lr = LogisticRegression(random_state = 4)
lr_accuracy, lr_recall, lr_precision, lr_f1_score = fit_n_print(lr, Orig_X_train, Orig_X_test, Orig_y_train, Orig_y_test)
#Printing Accuracy Score for model
print("\n Logistic Model Score:",lr_accuracy)
2. Support Vector Machine¶
In [31]:
from sklearn import svm
from sklearn.svm import SVC
svm = SVC(gamma=0.025, C=3)
#fit the model on orighinal raw data
svm.fit(Orig_X_train,Orig_y_train)
svm_accuracy, svm_recall, svm_precision, svm_f1_score = fit_n_print(svm, Orig_X_train, Orig_X_test, Orig_y_train, Orig_y_test)
print("\n Support Vector Machine Model Score:",svm_accuracy)
3. Navie Bayes¶
In [32]:
from sklearn.naive_bayes import GaussianNB
nb = GaussianNB()
nb_accuracy, nb_recall, nb_precision, nb_f1_score = fit_n_print(nb, Orig_X_train, Orig_X_test, Orig_y_train, Orig_y_test )
print("\n Naive Bayes Model Score:",nb_accuracy)
6. Applying Dimensionality Reduction using PCA¶
Performing PCA on given Dataset¶
We will perform PCA in following steps:¶
- Split our data into train and test data set
- normalize the tarining set using standard scalar
- Calculate the covariance matrix.
- Calculate the eigenvectors and their eigenvalues.
- Sort the eigenvectors according to their eigenvalues in descending order.
- Choose the first K eigenvectors (where k is the dimension we'd like to end up with).
- Build new dataset with reduced dimensionality.
- Split our data into train and test data set
In [33]:
#Now separate the dataframe into dependent and independent variables
X = Data.iloc[:,0:18].values
y = Data.iloc[:,18].values
- Normalize the tarining set using standard scalar
In [34]:
from sklearn.preprocessing import StandardScaler
sc =StandardScaler()
X = sc.fit_transform(X)
- Calculate the covariance matrix.
In [35]:
#Calculating Covarience Matrix
cov_matrix = np.cov(X.T)
print("cov_matrix shape:",cov_matrix.shape)
print("Covariance_matrix",cov_matrix)
- Calculating Eigen Vectors & Eigen Values: Using numpy linear algebra function
In [36]:
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
print("Eigen Vector \n%s", eigenvectors)
In [37]:
#Printing Eigen value
print("Eigne Value \n%s", eigenvalues)
- Sort the eigenvectors according to their eigenvalues in descending order.
In [38]:
# Make a set of (eigenvalue, eigenvector) pairs:
eig_pairs = [(eigenvalues[index], eigenvectors[:,index]) for index in range(len(eigenvalues))]
# Sort the (eigenvalue, eigenvector) pairs from highest to lowest with respect to eigenvalue
eig_pairs.sort()
eig_pairs.reverse()
print(eig_pairs)
# Extract the descending ordered eigenvalues and eigenvectors
eigvalues_sorted = [eig_pairs[index][0] for index in range(len(eigenvalues))]
eigvectors_sorted = [eig_pairs[index][1] for index in range(len(eigenvalues))]
# Let's confirm our sorting worked, print out eigenvalues
print('Eigenvalues in descending order: \n%s' %eigvalues_sorted)
In [39]:
tot = sum(eigenvalues)
var_exp = [( i /tot ) * 100 for i in sorted(eigenvalues, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
print("Cumulative Variance Explained", cum_var_exp)
plt.plot(var_exp)
Out[39]:
- We can observe that their is steep drop in variance explained with increase in number the number of Principal Components.
- Also, top 10 principal components contribute 98.5% variance (information). So, we can proceed further with 10 components.
Plotting The Explained Variance and Principal Components:¶
In [40]:
# Ploting
plt.figure(figsize=(8 , 7))
plt.bar(range(1, eigenvalues.size + 1), var_exp, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, eigenvalues.size + 1), cum_var_exp, where='mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
- From above we plot we can clealry observer that 8 dimension() are able to explain 95 %variance of data.
- so we will use first 8 principal components going forward and calulate the reduced dimensions.
Dimensionality Reduction¶
Now 8 dimensions seems very reasonable and With 8 variables we can explain over 95% of the variation in the original data!
In [83]:
# Using scikit learn PCA here. It does all the above steps and maps data to PCA dimensions in one shot
from sklearn.decomposition import PCA
# NOTE - we are generating only 10 PCA dimensions.
pca = PCA(n_components=10)
principalComponents = pca.fit_transform(X)
reduced_pca = pd.DataFrame(data = principalComponents)
reduced_pca.transpose()
Out[83]:
In [84]:
pca.components_
Out[84]:
In [85]:
#Let us check The Pairplot Of Reduced Dimension After PCA:
sns.pairplot(reduced_pca, diag_kind='kde')
Out[85]:
It is clealry visible from the pairplot above that:
- After dimensionality reduction using PCA our attributes have become independent with no correlation among themselves. As most of them have cloud of data points with no lienaer kind of relationship.
In [86]:
#PCA Data
pca_X_train,pca_X_test,pca_y_train,pca_y_test = train_test_split(reduced_pca,y,test_size=0.30,random_state=10)
In [87]:
# Since we will be using multiple models using the same data, thhe following is a user defined function for that purpose
def fit_n_print1(model, pca_X_train, pca_X_test, pca_y_train, pca_y_test): # take the model, and data as inputs
from sklearn import metrics
model.fit(pca_X_train, pca_y_train) # fir the model with the train data
pca_pred_train = model.predict(pca_X_train) # model predictions on the train data
pca_pred = model.predict(pca_X_test) # make predictions on the test set
score = round(model.score(pca_X_test, pca_y_test), 4) # compute accuracy score for test set
#Printing Confusion Matrix
cm = metrics.confusion_matrix(pca_pred,pca_y_test, labels = [0,1,2])
df_cm = pd.DataFrame(cm, index = [i for i in ["Van ", "Car ", "Bus"]],
columns = [i for i in ["Van ", "Car ", "Bus"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g')
print(classification_report(pca_y_test,pca_pred))
recall = metrics.recall_score(pca_y_test, pca_pred, average='macro')
precision = metrics.precision_score(pca_y_test, pca_pred, average='macro')
f1_score = round(2*precision*recall/(precision + recall),3)
return score, recall, precision, f1_score # return all the metrics
1. Logistic Regression After PCA¶
In [88]:
# Fitting Logistic Regression to the Training set
lr1 = LogisticRegression(random_state = 4)
lr1_accuracy, lr1_recall, lr1_precision, lr1_f1_score = fit_n_print1(lr1, pca_X_train, pca_X_test, pca_y_train, pca_y_test)
#Printing Accuracy Score for model
print("\n Logistic Model Score:",lr1_accuracy)
2. SVM after PCA¶
In [89]:
svm1 = SVC(gamma=0.025, C=3)
#fit the model on orighinal raw data
svm1.fit(pca_X_train,pca_y_train)
svm1_accuracy, svm1_recall, svm1_precision, svm1_f1_score = fit_n_print1(svm1, pca_X_train, pca_X_test, pca_y_train, pca_y_test)
print("\n Support Vector Machine Model Score:",svm1_accuracy)
In [90]:
#Use the SVM Classifier with k fold cross validation
from sklearn.model_selection import cross_val_score
scores = cross_val_score(svm1,reduced_pca , y, cv=10)
print(scores)
print('Average score: ', np.mean(scores))
3. Navie Bayes After PCA¶
In [91]:
nb1 = GaussianNB()
nb1_accuracy, nb1_recall, nb1_precision, nb1_f1_score = fit_n_print1(nb1, pca_X_train, pca_X_test, pca_y_train, pca_y_test )
print("\n Naive Bayes Model Score:",nb1_accuracy)
7. Apply Grid Search & Cross-Validation:To Tune Our Model and Validate The Model's Accuracy Score¶
In [92]:
import itertools
def classifiers_hypertune(name,rf,param_grid,x_train_scaled,y_train,x_test_scaled,y_test,CV):
from sklearn import metrics
CV_rf = GridSearchCV(estimator=rf, param_grid=param_grid, cv=CV, verbose= 1, n_jobs =-1 )
CV_rf.fit(x_train_scaled, y_train)
y_pred_train = CV_rf.predict(x_train_scaled)
y_pred_test = CV_rf.predict(x_test_scaled)
#model.fit(X_train_scaled, y_train)
print('Best Score: ', CV_rf.best_score_)
print('Best Params: ', CV_rf.best_params_)
score = round(accuracy_score(y_test, y_pred_test),4)
#Classification Report
print(name+" Classification Report: ")
print(classification_report(y_test, y_pred_test))
#Printing Confusion Matrix
cm = metrics.confusion_matrix(y_pred_test,y_test, labels = [0,1,2])
df_cm = pd.DataFrame(cm, index = [i for i in ["Van ", "Car ", "Bus"]],
columns = [i for i in ["Van ", "Car ", "Bus"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g')
recall = metrics.recall_score(y_test, y_pred_test, average='macro')
precision = metrics.precision_score(y_test, y_pred_test, average='macro')
f1_score = round(2*precision*recall/(precision + recall),3)
return score, recall, precision, f1_score # return all the metrics
4. KFold and GridSerach After PCA¶
In [93]:
from sklearn.model_selection import GridSearchCV
svmc1 = SVC()
#Let's See What all parameters one can tweak
print("SVM Parameters:", svmc1.get_params())
# Create the parameter grid based on the results of random search
param_grid = [
{'C': [0.01, 0.05, 0.5, 1], 'kernel': ['linear']},
{'C': [0.01, 0.05, 0.5, 1], 'kernel': ['rbf']},
]
param_grid_1 = [
{'C': [1, 10, 100, 1000], 'kernel': ['linear']},
{'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},
]
In [94]:
#K Fold Cross Validation Grid Search After PCA
svmc1_accuracy, svmc1_recall, svmc1_precision, svmc1_f1_score=classifiers_hypertune("Support Vector Machine",svmc1, param_grid,pca_X_train, pca_y_train, pca_X_test, pca_y_test,10)
print("\n SVM Model Score:",svmc1_accuracy)
KFold and GridSerach Before PCA¶
In [95]:
#K-Fold Cross Validation on Original Data
svmc = SVC()
svmc_accuracy, svmc_recall, svmc_precision, svmc_f1_score=classifiers_hypertune("Support Vector Machine",svmc, param_grid,Orig_X_train, Orig_y_train, Orig_X_test, Orig_y_test,10)
print("\n SVM Model Score:",svmc_accuracy)
8. Compare the accuracy scores and cross validation scores of Support vector machines¶
In [96]:
result = pd.DataFrame({'Model' : ['Before PCA Logistic Regression', 'After PCA Logistic Regression',"Before PCA Naive Bayes'", "After PCA Naive Bayes'", 'Before PCA SVM','After PCA SVM','Before PCA SVM with K Fold & Grid Serach', 'After PCA SVM with K Fold & Grid Search'],
'Accuracy': [ lr_accuracy , lr1_accuracy , nb_accuracy , nb1_accuracy , svm_accuracy, svm1_accuracy,svmc_accuracy,svmc1_accuracy],
'Recall' : [lr_recall , lr1_recall , nb_recall ,nb1_recall, svm_recall,svm1_recall,svmc_recall,svmc1_recall],
'Precision': [lr_precision , lr1_precision , nb_precision ,nb1_precision , svm_precision,svm1_precision,svmc_precision,svmc1_precision],
'F1 Score' : [lr_f1_score, lr1_f1_score, nb_f1_score, nb1_f1_score, svm_f1_score,svm1_f1_score,svmc_f1_score,svmc1_f1_score]})
result
Out[96]:
- From the above obesrvation we can see the comparision of model before and After applying PCA.
- It is clearly visible that SVM performed very well as compare to other model.
- After applying PCA it got reduced but after applying grid search and PCA result was improved.
- The model I would pick SVM as my final model because it has better numbers overall.
In [ ]: